Spark Core - Execution Engine: DAG Tracing Workbook
This workbook walks you through tracing execution plans and partition layouts.
1. Execution Flow
Analyze the following Spark operation pipeline:
[ FileScan (3 partitions) ] Map (Narrow) Filter (Narrow) reduceByKey (Wide - 2 Partitions)
2. Tasks
Task 1: Draw the DAG Stages
- Map the transformations to physical stages.
- Calculate the exact number of Tasks launched in Stage 1 and Stage 2.
Task 2: Trace Shuffle Boundaries
Trace where the shuffle write files are generated, how they are indexed, and how many network data transfers (shuffle fetches) occur.
3. Step-by-Step Solutions
Solution 1: Stage and Task calculations
- DAG Analysis:
STAGE 1 (3 Tasks) STAGE 2 (2 Tasks)
[ FileScan ] [ Map ] [ Filter ] [ Shuffle Write ] [ Shuffle Read ] [ reduceByKey ]
- Task Mapping:
- Stage 1: The initial stage handles narrow transformations. Because the input file is split into 3 partitions, Spark schedules 3 parallel Tasks (one per partition block).
- Stage 2: The shuffle boundary is reached at
reduceByKey. The target partitioning parameter is set to 2 partitions. Therefore, Spark compiles and schedules 2 parallel Tasks to process the shuffled partitions.
- Total Tasks: 5 tasks.
Solution 2: Shuffle Boundary Tracing
- Shuffle Write Files: Stage 1 has 3 tasks, so it generates 3 index files and 3 data files on the local scratch disks of the mapper executors.
- Shuffle Read Fetches: Stage 2 has 2 tasks. Each task must pull its assigned partition data from all 3 mapper files:
- Task 1 (Partition 0) fetches its segment from mapper 1, mapper 2, and mapper 3 (3 network fetches).
- Task 2 (Partition 1) fetches its segment from mapper 1, mapper 2, and mapper 3 (3 network fetches).
- Total Network Fetches: 6 fetches.
Find this content helpful?
☕ Buy me a coffee